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EXECUTIVE  SUMMARY 


This  report  summarizes  the  findings  of  a  three  year  research  program 
supported  by  a  grant  from  the  U.S.  Army  Research  Institute  to  assess  the 
effectiveness  of  short-term  training  programs  for  senior  national  security 
officials.  The  research  involved  a  twofold  thrust:  (1)  that  required  to 
develop  an  appropriate  model,  and  (2)  that  associated  with  assessing  model 
effectiveness . 

The  model  that  was  developed,  the  Harvard  Executive  Program  in  Nation¬ 
al  and  International  Security,  was  designed  specifically  to  enhance  the 
effectiveness  of  high  level  military  and  civilian  personnel  who  are  in  (or 
moving  into)  posts  where  their  personal  decisions  or  recommendations  can 
critically  affect  the  political,  economic,  or  military  interests  of  the 
United  States.  The  basic  intent  behind  the  program  was  to  provide  in¬ 
creased  understanding  of  selected  substantive  areas  and,  more  importantly, 
to  provide  tools  that  participants  would  find  useful  on  a  continuing  basis. 

An  evaluation  of  the  first  three  years  of  the  program  has  resulted  in 
the  following  conclusions: 

•  Short-term  training  along  the  lines  of  the  Harvard  model  can 
effectively  raise  the  levels  of  participants'  perceived  compe¬ 
tence  in  critical  areas  of  national  security  concern. 


Pre-Program 


Most  participants  continue  to  benefit  beyond  the  termination  of 
the  program  through  an  ongoing  application  of  newly  acquired 
skills  to  their  daily  occupations. 

Average  "Competence"  Across  All  Areas 
(Scale:  100  to  1,000) 


Post-Program  Six  Months  Post 


714  (+14%) 
714  (+22%) 
683  (+34%) 


741  (+4%) 
752  (+5%) 
733  (+7%) 


Those  entering  with  lower  levels  of  perceived  competence  tend  to 
gain  proportionally  more  than  their  colleagues  who  enter  with 
higher  levels  of  proficiency. 

There  has  been  an  increase  in  value  added  by  the  program  as  the 
trend  has  been  toward  accepting  applicants  with  lower  levels  of 
perceived  competence. 

The  program  has  brought  each  year's  group  to  essentially  the  same 
six  month  post-program  level  of  competence,  independent  of  their 
respective  pre-program  levels. 

War  College  graduates  characteristically  experience  marginally 
higher  gains  than  do  non-War  College  graduates.  The  same  obser¬ 
vation  holds  true  for  military  vs.  civilian  participants,  with 
the  former  realizing  the  greater  gains. 
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INTRODUCTION 


This  is  the  third  and  summary  report  on  the  effectiveness  of  short 
term  training  programs  for  senior  national  security  officials.  In  asses¬ 
sing  training  effectiveness,  research  was  conducted  on  two  fronts.  First 
there  was  the  research  required  to  develop  a  program  that  could  serve  as  a 
suitable  model,  i.e.  the  Harvard  Executive  Program  in  National  Security. 
Second  was  that  which  was  required  to  assess  how  effectively  the  program 
met  its  basic  objectives. 

The  research  to  support  the  program  itself  primarily  involved  the 
development  of  appropriate  case  studies.  As  the  program's  principal  peda¬ 
gogical  instrument,  the  case  study  represents  one  of  the  more  rewarding 
forms  of  research,  particularly  where  application  is  concerned.  Exposure 
to  high-level  officials  provides  an  immediate  and  demanding  test  of  ac¬ 
curacy  and  effectiveness  and,  through  the  feedback  that  ensues,  serves  to 
enrich  the  quality  of  the  case  materials.  This  has  certainly  proven  to  be 
the  case  with  those  studies  developed  to  date  (see  Attachment  A) . 

While  the  case-related  research  is  a  critical  ingredient  in  the  makeup 
of  the  program,  it  is  the  assessment  of  program  effectiveness  that  consti¬ 
tutes  the  principal  focus  of  this  report.  To  properly  understand  the 
approach  taken,  a  thorough  knowledge  of  the  program  background  becomes 
helpful. 

BACKGROUND 

As  the  panoply  of  government  problems  has  grown  increasingly  complex, 
the  need  for  improved  executive  development  in  the  public  sector  has  become 
more  readily  apparent.  The  very  manner  in  which  the  Federal  Service  is 
structured,  with  its  continual  turnover  of  political  appointees  in  most  of 


the  top  policy  jobs  (some  of  whom  are  qualified;  others  not)  tend;:  to 
foster  inefficiency  in  policy  design  and  execution. 

A  recent  survey  of  top  level  officials  within  the  national  security 
community  suggests  a  strong  need  to  improve  management  capability  in  both 
the  military  and  civilian  spheres.  On  the  military  side,  there  is  current¬ 
ly  very  little  available  in  the  way  of  continuing  education  at  the  "corpor¬ 
ate  executive  level."  While  there  is  a  plethora  of  training  programs  and 

opportunities  available  for  military  personnel  at  the  0-4  through  0-6 

levels  (Lieutenant  Commander  through  Captain;  Major  through  Colonel),  there 
is  very  little  beyond  that.  Each  of  the  Services  does  send  its  Flag  of¬ 
ficer  selectees  to  an  indoctrination  course  of  two  or  three  week's  dura¬ 
tion,  but  these  courses,  for  the  most  part,  have  an  internal  Service  rocus 

and  tend  to  be  somewhat  mechanical  in  nature. 

Although  little  in  the  way  of  executive  training  is  offered  for  Flag 
and  General  officers,  the  need  for  something  substantial  is  quite  real.  In 
some  respects,  the  term  "General"  ought  to  equate  to  "generalist  manager." 
In  many  instances,  however,  the  path  to  Flag  or  General  officer  rank  is 
quite  specialized;  and  once  there,  the  individual  can  serve  for  as  many  as 
ten  to  fifteen  years  with  virtually  no  opportunity  for  intellectual  or 
managerial  refurbishment.  Adding  to  the  problem  is  the  fact  that  there  are 
certain  areas  of  critical  importance  to  the  top  Service  manager  which  a.f 
not  covered  well  in  any  of  the  multitudinous  forums  to  which  he  or 
have  been  exposed  along  the  way.  Here  reference  is  made  specifically  to 
how  business  is  conducted  between  the  Department  of  Defense  (DOD)  and  other 
agencies  in  the  Executive  Branch;  the  interaction  between  the  Executive 
departments  and  the  Congress;  how  the  different  perspectives  of  the  Office 
of  the  Secretary  of  Defense  (OSD),  the  Joint  Chiefs  of  Staff  (JCS),  and  the 


Services  come  into  play,  what  is  involved/ required  in  getting  decisions 
implemented;  and  the  like.  The  situation  is  further  exacerbated  by  the 
inevitable  tendency  for  the  incumbent  (or  the  incumbent’s  superior)  to  feel 
that  the  job  demands  at  that  level  are  too  consuming  to  permit  extrication 
for  any  significant  period  of  training. 

The  needs  on  the  civilian  side  are  even  more  pressing.  In  May  of 
1974,  the  U.S.  Army  Management  Engineering  Training  Agency  (AMETA)  conduct¬ 
ed  a  "Study  of  Management  and  Executive  Development  in  Industry,  Universi¬ 
ties,  and  the  Federal  Government"  (13).  In  this  study,  the  authors  struck 
a  comparison  between  executive  development  efforts  in  the  private  and 
public  sectors  and  found  the  latter  wanting.  They  concluded  that  executive 
development  in  government  is  anything  but  institutionalized;  that  there  is 
little,  if  any,  accountability  by  senior  managers;  and  that  there  is  no 
system  for  identifying,  training,  and  tracking  prospective  executives.* 

More  recently,  in  March  of  1976,  a  panel  of  the  National  Academy  of 
Public  Administration  completed  a  study  entitled  "Strengthening  Civilian 
Executive  Development  in  the  Department  of  Defense"  (8).  Conducted  at  the 
request  of  the  Assistant  Secretary  of  Defense  (Manpower  and  Reserve  Af¬ 
fairs),  the  study  generated  a  number  of  recommendations  that  accommodated 
what  the  panel  found  at  that  time  to  be  the  general  characteristics  of 
Federal  Civil  Service  (i.e.,  limited  lateral  entry,  little  mobility  between 
agencies  or  occupational  groups,  and  virtually  no  "conscious  system"  for 
developing  executive  leadership)  while  specifically  addressing  the  unique 
problems  of  DOD.  In  this  regard,  the  panel  saw  the  need  for  consistent  top 


*  To  be  sure,  the  recent  advent  of  the  Senior  Executive  Service  with  its 
attendant  emphasis  on  management  has  helped  to  some  extent;  but  many  of  the 
observations  based  on  the  earlier  system  continue  to  hold  true. 
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level  support  of  executive  development  programs  as  the  most  critical  pro¬ 
blem  facing  the  Department.  Sheer  size  coupled  with  the  fact  that  most 
career  posts  in  Defense  are  occupied  either  by  political  appointees  or  the 
military  (with  attendant  brief  periods  of  incumbency)  makes  it  very  diffi¬ 
cult  to  personify  the  top  level  interest  that  is  required. 

The  National  Academy  study  also  pointed  out  that  over  the  five  year 
period  from  1976  through  1981,  approximately  50%  of  the  DOD  supergrade 
workforce  would  become  eligible  for  retirement.  Thus,  it  was  suggested 
that  a  primary  focus  of  DOD  executive  training  should  be  to  develop  the 
replacements  for  what  is  likely  to  be  an  inordinately  high  number  of  vacan¬ 
cies.  Related  to  this  suggestion  was  -he  fact  that  the  civilian  personnel 
system,  while  not  a  closed  system,  is  heavily  dominated  by  single  career 
personnel.  Indeed,  approximately  80%  of  DOD  supergrade  equivalent  jobs  are 
filled  from  within  the  Department.  The  inference  is  twofold:  (1)  it  is 
unrealistic  to  shortchange  executive  development  on  the  basis  that  poten¬ 
tial  candidates  have  already  received  such  training  in  private  industry 
prior  to  entering  government,  and  (2)  any  executive  development  program  for 
DOD  should  be  heavily  slanted  toward  meeting  the  specific  requirements 
associated  with  managing  the  national  security  process. 

In  view  of  the  above,  Harvard  University  decided  in  April  of  1978  to 
offer  a  two  week  Executive  Program  in  National  Security  on  an  annual  basis, 
with  the  first  program  to  take  place  in  August  of  the  same  year.  In  Jan¬ 
uary  1979,  the  Navy  Personnel  Research  and  Development  Center  published  a 
Navy  Civilian  Executive  Study  (8)  that  implicitly  endorsed  the  intent 
behind  the  Harvard  effort  (and  any  others  like  it): 


Although  relatively  few  executives  have  had  extensive  academic 
training  in  management,  leadership,  or  administration,  they  spend 
most  of  their  time  performing  tasks  in  these  areas.  This  high¬ 
lights  a  major  training  need.  It  includes ...  general  management 
knowledge  and  skills  (e.g.  decision-making,  communications) .. .A 
need  was  also  identified  for  the  integration  of  civilian  and 
military  training  in  the  shore  establishment. 

PROGRAM  GOALS 

Modeled  after  earlier  programs  pioneered  by  the  Business  School  for 
senior  business  executives,  the  Executive  Program  in  National  Security 
(subsequently  renamed  the  Executive  Program  in  National  and  International 
Security)  is  designed  for  Flag  and  General  rank  military  officers  and  for 
high-level  civilian  officials  who  are  either  in,  or  moving  into,  posts 
where  their  personal  decisions  or  recommendations  can  affect  the  critical 
political,  economic  or  military  interests  of  the  United  States.  Its  prin¬ 
cipal  goals  are  to  improve  participants’  understanding  of: 

differences  in  interest,  perspective  and  style  associated  with 
varying  types  of  responsibilities  in  differing  organizations; 
the  economic,  political,  technological,  and  organizational  con¬ 
text  within  which  national  security  policies  and  programs  are 
framed;  and 

various  dimensions  of  high-level  management,  including  the  uses 
and  misuses  of  formal  analysis  in  decision-making  and  policy 
design. 

While  the  principal  themes  of  the  program  include  management  perspec¬ 
tive,  policy  development,  and  uses  of  analysis,  it  is  the  dimension  of 
perspective  that  constitutes  the  central  thread  of  the  curriculum.  As 
noted  by  Gordon  Pask  of  System  Research  Ltd.  during  a  recent  ARI-sponsored 
conference  on  decision  making  in  complex  systems: 
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"There  are  and  there  must  be  multiple  perspectives  in 
order  to  have  decisions  at  all.  These  could  be  differ¬ 
ent  individuals  or  perspectives  entertained  by  one 
individual.  Individuals  are  not  very  relevant  in  this 
sense  because,  when  they  engage  in  activity,  perspec¬ 
tives  tend  to  become  distributed.  Individual  differ¬ 
ences,  if  one  recognises  that  they  are  not  context 
free}  are  important. . .Good  decisions  must  be  defined  in 
context.  Good  decision  making  involves  the  ability  to 
abduce  and  resolve  juxtaposed  abductions,  and  the 
ability  to  maintain  the  variety  of  the  system  while 
avoiding  chaos."  (11) 


In  getting  participants  to  think  critically  about  the  perspectives,  stakes, 
and  interests  of  their  counterparts  in  and  around  government,  it  becomes 
necessary  to  analyze  the  setting  in  which  problems  arise  and  the  particular 
incentives  which  various  actors  face.  Individuals  behave  differently  in 
different  organizational  settings  according  to:  (1)  their  own  history  and 
that  of  their  organization,  (2)  the  reward/sanction  structure  posed  by 
their  career  system,  and  (3)  which  of  the  many  faces  of  an  issue  they 
confront.  Thus,  a  key  goal  of  the  program  is  to  increase  the  participants' 
awareness  and  understanding  of  the  respective  frames  of  reference  of  the 
Presidency;  the  Congress;  the  media;  political  appointees;  careerists, 
whether  they  be  military  or  civilian;  the  private  sector;  and  other  coun¬ 
tries.  These  categories  were  chosen  as  representative  of  the  work  areas 
germane  to  national  security  policy  development  and  program  implementation. 


COURSE  DESCRIPTION 

The  pedagogical  approach  used  in  the  program  is  highly  participatory 
in  nature.  Because  those  government  and  private  sector  executives  who 
attend  collectively  share  extraordinary  experience  and  wisdom,  the  curri¬ 
culum  is  designed  so  that  participants  derive  maximum  benefit  not  only  from 
the  faculty,  but  from  one  another.  Case  studies,  supplemented  by  seminars 
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and  some  lectures,  provide  intense,  vicarious  experience  in  a  broad  spec¬ 
trum  of  managerial  situations  and  increased  insight  into  a  wide  range  of 
important  security-related  issues.  These  typically  include  such  topics  as 
global  economic  forces  and  the  determinants  of  national  economic  policy, 
regional  problems  and  arrangements,  constraints  that  shape  Presidential  and 
departmental  budgets,  the  interplay  between  foreign  and  domestic  policy, 
political  assessment  and  forecasting,  uses  and  misuses  of  history,  resource 
allocations  and  constraints,  executive-legislative  relations,  government- 
press  relations,  and  civilian-military  relations. 

It  is  intended  that  participants  will  leave  not  only  with  a  better 
understanding  of  certain  critical  issues  of  national  and  international 
security  policy,  but  also  with  a  keener  appreciation  of  the  reasons  for 
wide  differences  in  perception  and  position  relating  to  such  issues  and  an 
enhanced  ability  to  devise  and  implement  effective  solutions  to  complex  and 
multifaceted  problems. 

The  program  is  tailored  to  meet  specific  needs  which  senior  officers 
and  officials  have  identified  as  lacking  in  their  own  preparation  for 
assuming  posts  at  the  highest  levels  in  the  national  security  community. 
It  is  a  total  immersion  experience  with  a  typical  day  starting  at  7:00  a.m. 
and  ending  late  that  night  with  preparations  for  the  following  day's  clas¬ 
ses.  During  the  course  of  the  day,  the  participants  are  exposed  to  three 
90-minute  classes,  and  an  hour  of  group  discussion.  There  are  a  total  of 
30  such  class  sessions  during  the  program,  taught  by  leading  experts  in 
their  fields.  (See  Attachments  B  and  C  for  typical  listings  of  assignments 
and  faculty,  respectively.)  To  facilitate  informal  interaction  among 
participants,  a  number  of  social  and  athletic  events  are  also  scheduled. 


Since  the  notion  of  perspective  is  such  a  critical  ingredient  of 
program  substance,  considerable  effort  is  expended  in  achieving  balanced 
representation  from  all  segments  of  the  national  security  community.  See 
Attachment  D  for  a  typical  roster  of  program  participants. 

ASSESSMENT 

The  analytical  approach  to  the  assessment  of  program  effectiveness  has 
remained  essentially  the  same  over  the  first  three  years  of  the  program, 
comparing  respectively  the  responses  of  military  vs.  civilian  and  War 
College  vs.  non-War  College  participants.  Highlighted  in  this  vein  is  an 
in-depth  analysis  of  the  features  and  patterns  of  gain  in  each  partici¬ 
pant's  perceived  competence  in  dealing  with  the  various  dimensions  of 
national  and  international  security.  Cluster  and  correlation  analysis 
techniques  were  introduced  for  this  purpose  during  the  second  year  of  the 
program.  This  approach  has  been  carried  forward  this  year  as  well. 

A  comparison  of  the  standard  parameters  of  program  assessment  for  the 
three  years  is  presented  in  Table  I. 


TABLE  I 


A.  On  a  scale  of  1  (ineffective)  to  5  (extremely  effective): 

1978  1979  1980 


1. 

Overall  usefulness 

4.3 

4.6 

4.4 

2. 

Instructors 

4.1 

4.3 

4.1 

3. 

Cases 

4.0 

4.1 

4.0 

4. 

Administration 

4.8 

4.9 

4.7 

B.  With  respect  to: 


Program  expectations 


Exceeded 
Equaled 
Fell  Short 


8  26  28 

20  18  41 

1  0  4 
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Work  load 


Too  heavy 
About  right 
Too  light 

Program  length 

Too  long 
About  right 
Too  short 


1  10  23 

29  34  51 

1  0  0 


1  1  3 

29  36  64 

1  7  5 


In  terms  of  achieving  the  principal  objective  of  increasing  student  under¬ 
standing  of  the  perspectives  of  other  participants  in  the  national  security 
process,  an  average  grade  of  4.5  was  assigned  during  the  third  year.  This 
compares  with  4.2  and  4.4  respectively  for  the  two  previous  years.  Each 
year  this  score  was  refined  through  a  "ratio  scaling"  technique  wherein 
each  executive  was  asked  to  evaluate  on  a  scale  of  100  to  1000  the  degree 
to  which  he  or  she  felt  competent  to  deal  with  issues  in  each  of  the  prin¬ 
cipal  areas  of  interest  (see  Attachment  E).  In  this  regard,  a  series  of 
three  data  points  were  taken:  one  prior  to  the  program,  one  immediately 
following  completion  of  the  program,  and  one  six  months  later  (after  suffi¬ 
cient  time  had  elapsed  to  permit  meaningful  application  of  what  was  learn¬ 
ed).  In  view  of  the  heavily  subjective  nature  of  the  total  evaluation 
scheme,  however,  the  pre-program  data  was  taken  at  the  same  time  as  that 
immediately  following  the  program.  This  was  done  to  help  ensure  greater 
consistency  in  standards,  i.e.,  the  participants  may  not  have  fully  appre¬ 
ciated  at  the  outset  how  weak  (or  strong)  they  were  in  a  given  area  as  they 
were  after  they  had  completed  the  program.  It  also  had  the  added  benefit 
of  avoiding  an  artificial  "ceiling  effect"  wherein  one  might  have  given 
oneself  the  highest  rating  possible  at  the  beginning  of  the  program  and, 
after  learning  considerably  more  during  the  program,  then  been  forced  to 
use  the  same  (but  subsequently  misleading)  rating. 
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While  the  method  for  assessing  program  effectiveness  is  highly  subjec¬ 
tive  in  nature,  such  an  approach  is  dictated  by  the  sensitivities  involved 
in  dealing  with  participants  at  this  level.  Their  professional  status  is 
such  as  to  preclude  the  possibility  of  canvassing  their  superiors  for 
"before  and  after"  impressions  of  performance.  Attempts  at  evaluating 
classroom  performance  (including  videotape  analysis)  proved  equally  futile 
since  individual  participation  in  a  class  size  of  40  is  too  intermittent  to 
provide  a  reliable  indication.  Another  aspect  complicating  an  objective 
evaluation  of  participant  performance  is  the  duality  of  the  learning  pro¬ 
cess.  As  Ernest  May,  Faculty  Chairman  for  the  Program,  has  noted,  "the 
faculty  learn  as  much  from  the  experience  as  the  students." 

A  trait  common  to  most  high  level  officials  in  evaluating  the  qualita¬ 
tive  aspects  of  any  training  program  is  that  of  scrupulous  honesty.  For 
one  thing,  they  feel  an  inherent  sense  of  responsibility  toward  those  who 
attend  succeeding  programs  to  provide  the  most  candid  assessment  possible, 
even  when  it  comes  to  evaluating  themselves.  For  example,  it  is  not  at  all 
unusual  for  a  participant  to  evaluate  his  or  her  pre-program  understanding 
of  a  given  area  at  a  level  of  300  (on  the  100  to  1000  scale). 

As  indicated  in  Table  II,  the  average  participant  entered  the  program 
with  a  mean  "competence"  of  509  across  all  areas,  i.e.,  the  "average  parti¬ 
cipant"  felt  about  51%  as  competent  or  comfortable  in  an  "average  area"  as 
he  or  she  did  in  an  area  with  which  they  were  thoroughly  familiar  and  in 
which  they  felt  well  qualified  (see  Attachment  F),  During  the  course  of 
the  program,  this  mean  index  increased  to  683,  a  34%  improvement  over  the 
pre-program  level.  Six  months  later,  it  had  increased  an  additional  7%  to 
733.  Comparable  figures  for  the  previous  years  are  included  in  Table  II 
and  displayed  in  Figures  I  and  II. 
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TABLE  II 

Average  "Competence"  Across  All  Areas 


Pre- 

Post- 

Six 

Program 

Program 

Months  Post 

Total  Group 

(N=29) 

1978 

619 

714  (14%) 

741  (4%) 

Total  Group 

(N=30) 

1979 

589 

714  (22%) 

752  (5%) 

Total  Group 

(N=4l) 

1980 

509 

683  (34%) 

733  (7%) 

There  was  a  significant  decrease  in  the  pre-program  figures  over  the  first 
three  years  of  the  program.  As  Table  II  indicates,  the  pre-program  level 
dropped  from  619  in  1978  to  509  in  1980.  However,  the  six  months  post¬ 
program  levels  were  approximately  equal  for  each  year's  respondents.  This 
decrease  in  pre-program  levels  may  be  at  least  partially  attributable  to  a 
trend  towards  increased  specialization  in  the  represented  population.  For 
example,  there  was  greater  representation  from  the  technical  research  and 
development  community  in  each  succeeding  year.  The  fact  that  all  groups 
reached  approximately  the  same  post-program  levels  suggests  that  the  pro¬ 
gram  is  proving  a  certain  effectiveness  in  bringing  all  groups  to  the  same 
level  of  competence,  independent  of  their  perceived  pre-program  levels. 

More  revealing  are  the  increases  for  each  of  the  specific  areas  of 
emphasis.  Table  III  and  Figures  IIIA  and  IIIB  provide  the  indices  for  the 
1980  participants  in  the  seven  areas  of  major  interest. 


TABLE  III 

Competence  in  Specific  Areas  (1980) 


Pre-Program 

Gain  During 

Gain  in 

Area 

Level 

Program 

Six  Months 

Presidency 

451 

232 

58 

Congress 

502 

180 

67 

Politicians 

548 

146 

37 

Careerists 

688 

62 

57 

Media 

436 

194 

44 

Private  Sector 

482 

171 

40 

International 

453 

232 

50 
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PRESIDENCY  CONGRESS  POLITICIANS  CAREERISTS  KDIA  PRIV.  SECTOR  INTERNATT 


COMPETENCE  IN  SPECIFIC  AREAS  (1980) 


PRESIDENCY  CONGRESS  POLITICIANS  CAREERISTS  KDIA  PRIY.  SECTOR  INTERNAT'L 


AVERAGE  COMPETENCE-ALL  AREAS 


FIGURE 


Figure  IIIA  shows  that  the  1980  participants  felt  more  competent  in  dealing 
with  the  "careerist"  perspective  that  any  other.  This  comes  as  no  surprise 
since  the  majority  of  the  participants  are  either  military  or  civilian 
careerists  themselves.  As  might  be  expected,  they  experienced  little 
growth  in  this  area  as  a  result  of  the  program.  This  is  symptomatic  of  a 
broader  pattern:  the  higher  the  pre-program  competence,  the  less  the  gain 
during  the  program.  Figure  IIIB  indicates  that  by  the  end  of  the  program, 
all  competence  levels  were  close  to  equal;  with  the  largest  disparity 
occurring  between  the  Media  and  Careerist  categories. 

This  pattern  is  not  unique  to  the  1980  group.  Table  IV  provides  a 
comparison  of  the  gains  in  each  of  the  categories  common  to  all  three  years 
(the  areas  of  emphasis  changed  slightly  during  the  second  year;  thus  there 
is  no  1978  equivalent  of  the  Presidency  and  International  categories). 
Again,  as  might  be  expected,  the  high  pre-program  levels  for  Careerists, 
contrasts  with  those  of  the  lowest,  Private  Sector  and  Media.  By  the  six 
months  post-program  point,  all  areas  were  approximately  equal.  Fig¬ 
ures  IVA,  B,  and  C  further  illustrate  this  point. 


TABLE  IV 

Sample  Group  Comparisons 

Six  Months 

Pre-Program  Program  Gain  Post-Program  Gain 


1978 

1979 

1980 

1978 

1979 

1980 

1978 

1979 

1980 

Careerists 

699 

781 

688 

46 

38 

62 

22 

41 

57 

Congress 

615 

591 

502 

105 

113 

180 

38 

54 

67 

Politicians 

612 

624 

548 

111 

96 

146 

30 

44 

37 

Private  Sector 

601 

495 

482 

96 

153 

171 

32 

16 

40 

Media 

589 

536 

436 

88 

154 

194 

28 

46 

44 

It  is  noteworthy  that  there  are  no  glaring  differences  among  the  three 
groups.  Aside  from  the  general  downward  trend  in  pre-program  scores, 


THREE- YEAR  COMPARISONS 


POLITICIANS  PRIY.  SECTOR  *DIA 


CAREERISTS  CONGRESS  POLITICIANS  PRIY. SECTOR  HEDIA 


FIGURE  I VC 


ultimate  levels  achieved  in  each  of  the  five  common  areas  has  not  varied 
much.  Hence,  there  is  no  reason  to  believe  that  the  three  groups  were  not 
comparable  in  their  reactions  to  the  program,  nor  that  the  conclusions 

based  on  the  responses  from  one  group  are  not  generally  applicable  to 
others . 

Returning  to  Table  IV,  we  specifically  note  that  the  average  1980 
pre-program  level  is  about  80  less  than  the  corresponding  1979  level.  The 

only  area  which  is  not  significantly  lower  in  1980  is  Private  Sector, 

probably  because  it  is  the  single  area  that  is  most  uniformly  foreign  to 

government  participants.  Except  for  the  areas  of  Private  Sector  and  Career¬ 
ists,  the  average  "during  program"  gain  is  approximately  50  points  higher 
in  1980  than  in  1979.  The  reason  for  lesser  Private  Sector  gain  is  probab¬ 
ly  due  to  the  fact  that  the  pre-program  levels  were  approximately  the  same 
to  begin  with.  The  lack  of  growth  in  Careerists  is  probably  ue  to  x 
ceiling  effect. 

In  comparing  the  reactions  of  those  participants  in  the  1980  program 
who  had  previously  attended  War  College  with  those  who  had  not,  the  former 
entered  the  program  feeling  less  competent  (468  vs.  531  as  opposed  to  the 
1979  and  1978  experience  of  647  vs.  594  and  647  vs.  594  respectively)*  and 
left  feeling  that  they  had  gained  more  (212  vs.  153  as  was  also  true  in 
prior  years:  128  vs.  127  for  1979  and  103  vs.  91  in  1978).  Figure  V 


*  Why  the  War  College  participants  evaluated  themselves  at  a  lower  average 
pre-program  level  than  their  non-War  College  colleagues  during  the  third  yer 
defies  precise  explanation.  It  may  have  to  do  with  the  fact  that  program 
attendance  almost  doubled  between  the  second  and  third  years.  Since  this 
expansion  involved  the  acceptance  of  a  number  of  less-experienced  Flag  offi¬ 
cers  (of  which  the  typical  War  College  class  is  mostly  comprised),  this  may 
have  had  a  dampening  effect  on  the  self-perceived  pre-program  index  of  compe¬ 
tence  for  the  War  College  group  (and  the  military  group  as  reflected  in  Fig¬ 
ure  VI). 
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illustrates  this  comparison  for  the  last  two  years  of  program  data.  Thus 
it  appears  that  the  War  College  graduates  eventually  obtained  a  marginally 
higher  level  throughout  than  did  their  non-War  College  peers.  This  average 
level  gain  does  not  reflect  the  true  effect  of  the  program  on  these  two 
groups,  however. 

A  t-test  for  differences  between  means  was  applied  to  the  War  College 
and  non-War  College  group  averages  in  Table  VA  to  determine  whether  the 
differences  were  statistically  significant  (Attachment  G  contains  a  discus¬ 
sion  of  the  use  of  t-tests  for  this  analysis). 

TABLE  VA 

War  College  vs.  Non-War  College  (1980) 

Pre-Program  Post  Program  Six  Months 


War  Coll 

Non- War 

War  Coll 

Non-War 

War  Coll 

Non-War 

Presidency 

449 

452 

694 

677 

764 

729 

Congress 

473 

520 

683 

683 

780 

733 

Politicians 

433  * 

613 

678 

702 

763 

712 

Careerists 

604  ** 

737 

703 

777 

805 

808 

Media 

422 

444 

647 

621 

703 

658 

Private 

463 

492 

665 

646 

732 

671 

International 

435 

463 

687 

685 

767 

717 

Average 

468 

531 

680 

684 

759 

718 

War  College  =  15  respondents 

Non-War  College  =  26  respondents 

*  significantly  different  @  5%  level 

**  significantly  different  @  10%  level 


A  related  phenomenon  of  the  expansion  in  class  size  was  a  perceptible 
diminishment  in  camaraderie.  This  was  reflected  in  a  noticeable  decrease  in 
the  number  of  participants  who  responded  to  the  six  month  post  program  ques¬ 
tionnaire. 
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The  only  area  in  which  the  pre-program  difference  proved  significant  was 
that  of  Politicians,  wherein  War  College  graduates  rated  themselves  signi¬ 
ficantly  lower  than  did  non-War  College  attendees  (433  vs.  613).  The  two 
means,  433  and  613,  are  significantly  different  at  a  95%  confidence  level. 
Using  the  equations  in  Attachment  F: 


War  College  (WC) 

Non-War  College  (NWC) 

X 

433.3 

613.5 

S 

247 

207 

N 

15 

26 

.  (N  -1)S2  +  (N  -1)S2 

m.  ,  ,  /f.2.  _  wc  wc  v  nwc  nwc 

The  pooled  variance  (S  )  =  - ^ n  —  7» - ij) - 

1  wc  J  '“nwc 


=  49,103. 


The  sample  variance  for  differences  of  means 


N 

nwc 


=  5,162. 


The  t-statistic  (t-j) 
a 


=  2.51  with  degrees  of  freedom  39. 

t  =  2.51  is  significant  at  a  .016  level  (which  means  that  two  random 
samples  from  the  same  normal  distribution  will  have  a  t-statistic  as  large 
as  2.51  only  1.6%  of  the  time).  Note:  the  standard  deviations  above  are 
equal:  that  is,  S  f  S  is  rejected  at  the  5%  level. 

Table  VB  makes  a  similar  comparison,  except  the  respondents  are  sepa¬ 
rated  according  to  "military"  or  "civilian"  status.  Once  again,  the  only 
area  which  demonstrated  a  5%  significant  pre-program  difference  was  Politi¬ 


cians. 


TABLE  VB 

Competence  in  Specific  Areas  (1980) 
Military  vs.  Civilian 


Area 

Pre-Program 

Level 

Post-Program 

Level 

Six  Months 
Post  Level 

MIL 

CIV 

MIL 

CIV 

MIL 

CIV 

Presidency 

430 

458 

651 

695 

727 

747 

Congress 

481 

510 

685 

682 

777 

740 

Politicians 

436* 

588 

643 

712 

718 

735 

Careerists 

669 

695 

726 

758 

830 

798 

Media 

503 

412 

727** 

595 

745** 

648 

Private  Sector 

495 

477 

707 

633 

766** 

667 

International 

434 

460 

705 

678 

741 

733 

Average 

493 

514 

692 

679 

758 

724 

198 

Average 

573 

610 

717 

709 

752 

753 

191 

Military  =11  respondents 
Civilian  =  30  respondents 

*  significantly  different  @  5%  level 
**  significantly  different  @  10%  level 


As  can  be  seen  in  Table  VI,  the  non-War  College  respondents  gained 
significantly  less  (in  a  statistical  sense)  than  War  College  graduates  in 
the  Politicians  area  only. 


TABLE  VI 

During  Program  Gain 
War  College  vs.  Non-War  College 


Presidency 

220 

239 

Congress 

196 

172 

Politicians 

209 

113 

Careerists 

77 

54 

Media 

234 

174 

Private  Sector 

198 

157 

International 

230 

233 
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There  is  no  apparent  significance  to  this  difference  except  to  note  that 
the  only  area  in  which  the  military  respondents  also  significantly  out- 
gained  their  civilian  counterparts  was  that  of  Politicians  as  well. 


TABLE  VII 

Military 

vs.  Civilian 

Presidency 

249 

222 

Congress 

221 

154 

Politicians 

229 

92 

Careerists 

93 

42 

Media 

217 

180 

Private  Sector 

198 

154 

International 

248 

222 

This  is  counter  to  the  previous  year's  experience  in  which  the  non-War 
College  group  outgained  the  War  College  group  in  all  areas  except  Media, 
Private  Sector  and  International.  For  the  record,  in  1979,  the  military 
outgained  the  civilians  in  all  areas.  Figure  VI  compares  the  1980  results 
with  those  for  1979. 

Cluster  analysis  (5,6)  was  applied  to  the  1980  response  data  to  deter¬ 
mine  if  any  specific  identifiable  subset  of  attendees  benefitted  signifi¬ 
cantly  more  than  others  who  attended.  This  technique  was  initially  applied 
last  year  to  determine  which  participants  had  similar  gain  patterns  in  each 
of  the  seven  areas  of  emphasis.  Once  the  subsets  of  similar  scoring  parti¬ 
cipants  were  identified  (these  subsets  are  called  "clusters"),  then  the 
individuals'  backgrounds  (career  area,  service,  experience,  etc.)  were 
compared  to  determine  whether  there  were  any  significant  commonalities. 
These  commonalities  must,  of  course,  be  unique  to  the  cluster  so  as  to  set 
it  apart  from  other  clusters.  If  such  commonalities  are  found  to  exist, 
then  it  may  be  inferred  that,  inadvertently  or  not,  the  program  favors  a 
certain  class  of  attendee.  Thus,  cluster  analysis  indicates  among  which 
individuals  a  common  classification  should  be  sought. 
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Summarizing  the  results  of  the  cluster  analysis  of  the  1979  data:  two 
small  tightly  packed  clusters  and  three  distinct  "outliers"  were  discerned. 
Comparing  the  backgrounds  of  the  participants  in  the  two  clusters  and  the 
three  outliers  yielded  the  following: 

1.  There  were  no  significant  elements  of  commonality  in  the  two 
clusters . 

2.  The  three  "outliers"  had  backgrounds  similar  to  many  who  were 
much  closer  to  the  clusters. 

3.  The  average  distance  from  case  to  case  among  all  the  cases  was 
sufficiently  large  to  conclude  that  the  cluster  analysis  had  not 
highlighted  an  identifiable  subgroup  of  attendees  who  drew  inor¬ 
dinate  benefit  (or  obtained  no  benefit  at  all)  from  the  program. 

The  same  analytical  technique  (in  fact,  the  same  computer  program)  was 
applied  to  the  "during  program"  gain  data  for  the  41  respondents  from  the 
1980  program  with  nearly  identical  results  (see  Attachment  H  for  cluster 
analysis  algorithm  and  computer  analysis).  Instead  of  the  two  tightly 
packed  clusters,  there  were  two  more  loosely  associated  groups,  but  again, 
with  widely  varying  backgrounds.  There  were  six  outliers,  half  of  whom 
were  military  personnel,  with  two  of  them  having  had  War  College  experi¬ 
ence.  It  was  therefore  concluded  (even  more  strongly  than  it  was  in  1979) 
that  the  executive  program's  method  and  curriculum  are  not  biased  towards 
or  away  from  any  specific,  identifiable  subset  of  participants. 

As  explained  in  last  year's  report,  correlation  analysis  (4,5)  is  used 
to  quantify  the  degree  of  association  among  scores,  i.e.,  if  an  attendee 
gains  a  great  deal  during  the  program  in  area  A,  can  he  or  she  be  expected 
to  gain  significantly  in  area  B?  For  clarification:  the  charts  and  tables 
comparing  mean  competence  levels  and  mean  gain  levels  among  different  years 
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and  groups  indicate  how  well  different  participants  have  responded  to  the 
program  curriculum.  The  cluster  analysis  indicates  whether  a  particular 
subset  of  attendees  benefits  more  from  (or  responds  better  to)  the  program 
curriculum;  in  other  words,  it  addresses  the  program  design.  Correlation 
analysis,  on  the  other  hand,  indicates  how  consistently  individual  partici¬ 
pants  respond  to  the  seven  different  areas  of  emphasis  (Attachment  I  con¬ 
tains  a  mathematical  description  of  the  correlation  statistics). 

Recalling  the  results  of  the  correlation  analysis  on  the  1979 
data:  the  scores  in  the  Presidency  category  served  as  a  good  proxy  for  all 
gain  scores  for  an  individual;  that  is,  the  program  gains  for  Presidency 
correlated  highly  with  all  other  areas,  however,  there  did  not  appear  to 
be  any  synergistic  effects,  in  which  training  in  one  area  was  seen  as 
perfectly  complementing  that  in  another  such  that  there  were  inordinate 
gains  in  both,  or  that  training  in  one  area  supplemented  that  in  another  so 
well  that  equal  gains  would  be  realized  in  both  (such  effects  would  have 
been  made  apparent  by  singularly  high  correlations  in  one  or  two  cells  of 
the  matrix,  with  relatively  low  levels  elsewhere).  In  general,  there 
appeared  to  be  an  overall  complementary  tffect  to  the  training. 

The  analysis  has  been  extended  for  the  1980  data.  Correlation  sta¬ 
tistics  have  been  computed  and  compared  for  both  pre-program  levels  and 
during  program  gains  for  the  military  vs.  civilian  and  War  College  vs. 
non-War  College  groups.  Tables  VIII  a-h  present  the  individual  correlation 
matrices . 

Looking  at  the  civilian  vs.  military  comparison  and  focusing  on  the 
pre-program  levels  (Tables  a  and  c),  it  is  obvious  that  the  military  atten¬ 
dees'  responses  correlate  much  higher  in  all  areas  (with  the  exception  of 
the  "Congress")  than  do  their  civilian  colleagues,  and  lower  in  very  few. 
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CORRELATION  MATRIX 

TABLE  VIII  B 


CORRELATION  MATRIX 

TABLE  VIII  E 


CORRELATION  MATRIX 

TABLE  VIII  F 


CORRELATION  MATRIX 

TABLE  VIII  G 


Tables  e  and  g  provide  the  same  comparison  for  the  non-War  College  vs.  War 
College  with  the  latter  category  correlating  much  higher  than  the  former. 
These  higher  correlations  indicate  that  the  military/War  College  attendees 
are  more  consistent  in  their  responses  than  are  the  civilian/non-War  Col¬ 
lege  participants.  There  are  two  phenomena  at  work  here.  One  is  that  the 
military  and  War  College  experiences  tend  to  impart  similar  biases  and 
points  of  view  to  those  involved.  The  other  is  that  some  military  person¬ 
nel,  depending  on  their  assignments,  tend  to  consider  the  Congress  off- 
limits,  whereas  civilians  feel  no  such  inhibition.  Thus  we  see  that  the 
military  and  War  College  groups  tend  to  be  more  single-minded.  While  they 
don't  necessarily  score  higher  than  the  others,  they  do  score  with  greater 
consistency.  There  is  a  similar,  but  less  marked,  pattern  appearing  in  the 
"during  program"  gain  correlation  matrices.  The  similar  correlations 
between  the  military  and  War  College  groups  are  consistent  with  what  one 
would  expect  since  the  vast  majority  of  War  College  attendees  are  members 
of  the  Armed  Forces. 

EXPLORATION  OF  POSSIBLE  METHODOLOGIES 

During  the  first  year  of  the  program,  three  separate  areas  of  study 
were  subjected  to  mathematical  or  statistical  analysis,  each  of  which 
related  to  the  overall  objective  of  determining  training  effectiveness. 
The  investigation  subjects  included: 

1 .  Analysis  of  measures  of  effectiveness. 

The  investigations  in  this  area  were  directed  toward  assessment  of 
participant  responses  to  increase  their  perceptions  of  how  the  program 
had  affected  them.  The  analysis  involved  the  use  of  simple  descrip- 


tive  statistics  (means,  variances,  etc.)  for  each  of  the  program 
subject  areas,  trend  analysis  over  the  three  years,  determination  of 
subpopulation  scoring  patterns,  and  the  like.  These  investigations 
proved  most  fruitful  and  were  continued  in  subsequent  years. 

2.  Determination  of  long-term  effects. 

There  was  a  clear  trend  of  growth  in  all  areas  of  competence  both 
during  the  program  and  for  the  six  month  period  following  its  comple¬ 
tion.  The  dimensions  addressed  in  this  area  of  investigation  in¬ 
cluded:  (1)  how  long  the  program's  influence  would  be  felt  by  the 
participants,  and  (2)  to  what  levels  of  competence  they  could  be 
expected  to  rise.  Attempts  to  model  the  growth  pattern  as  a  non¬ 
stationary  stochastic  process  were  not  particularly  successful. 
Additional  data  beyond  that  collected  at  the  six  month  p*ost-program 
point  would  have  been  required  to  yield  conclusive  results.  Since  it 
was  felt  that  any  further  demands  on  participants'  time  in  this  regard 
would  have  represented  an  imposition,  this  area  of  investigation  was 
discontinued. 

3.  Prediction  of  applicant  population. 

Another,  less  direct,  measure  of  effectiveness  was  the  degree  to  which 
the  program  was  capable  of  attracting  an  increased  number  of  appli¬ 
cants  in  successive  years.  A  theoretical  model  of  the  application/ 
attendance  process  was  developed.  However,  since  the  model  was  based 
on  the  then  current  methods  of  recruiting,  which  changed  dramatically 
over  the  succeeding  years,  its  utility  as  a  predictive  tool  was  judged 
insignificant. 
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Methodologies 


1 .  Measures  of  Effectiveness. 

The  basic  approach  to  analyzing  measures  of  effectiveness  was  to 
identify  descriptive  statistics  that  characterized  the  patterns  of 
competency  scores  (12).  Other  than  the  simple  histograms  and  time 
plots  shown  in  this  report,  statistical  comparisons  have  been  ad¬ 
dressed  among  such  groups  as  military  vs.  civilian,  War  College  vs. 
non-War  College,  1979  vs.  1980,  and  area  vs.  area  with  such  analytic 
tools  as  t-tests,  F-ratio  tests,  correlation  analyses,  and  cluster 
analyses.  There  are,  of  course,  many  other  analytical  methodologies 
that  could  have  been  employed.  Instead  of  the  simple  correlations, 
covariance  analyses,  regression  analyses,  and  the  like  could  have  been 
performed.  In  addition  to  cluster  analysis,  factor  analysis  (3,5), 
logit  and  probit  analysis  (10),  discriminant  analysis  (2,3),  and 
principal  components  analysis  (2,5)  could  have  been  performed.  How¬ 
ever,  the  correlation  and  cluster  techniques  provide  intuitive 
results,  even  for  the  non-analyst.  Most  of  the  other  techniques  would 
have  required  manipulation  and  transformation  of  the  data,  often  in  a 
sense  that  would  probably  bury  interpretive  results.  Note  that  a 
rigorous  experimental  design  was  not  employed  wherein  it  was  decided 
ahead  of  time  to  analyze  the  effectiveness  measure  with  principal 
component  methodologies,  for  example,  and  then  construct  a  question¬ 
naire  to  provide  the  required  parameters  of  such  a  model. 

The  questionaire  that  was  used  was  designed  to  give  maximum  informa¬ 
tion  on  the  respondents'  opinions  of  the  program  and  its  effectiveness 
and  on  their  associated  personal  growth  in  competency  in  specific 
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areas  of  national  security  concern.  Preliminary  analysis  of  these 
data  indicated  that  there  may  have  been  some  underlying  structure  to 
the  participant  scores.  Hence  the  comparisons  of  military  vs.  civil¬ 
ian,  etc.  Note,  again,  most  of  the  methodologies  mentioned  above  have 
multiple  independent  variable  and  single  dependent  variable  models. 
We  did  not  have  such  a  structure  to  work  with.  Instead,  the  indivi¬ 
dual  scores  had  to  be  considered  as  dependent  variables,  while  civil¬ 
ian  or  military  or  War  College  experience,  etc.,  was  the  sole  indepen¬ 
dent  variable  available.  [Some  factor  analysis  techniques  could  have 
been  applied,  using  military/civilian  as  one  independent  variable  and 
War  College/non-War  College  as  another;  however,  none  of  the  models 
can  provide  meaningful  results  when  there  are  more  dependent  than 
independent  variables.  Hence  one  would  have  had  to  select  at  most  two 
scores  as  dependent  variables  or  build  some  sort  of  composite  depen¬ 
dent  variable  for  each  participant.  It  is  such  manipulation  and 
transformation  mentioned  above  that  would  fog  the  issues  —  it  just 
doesn't  make  good  mathematical/engineering  sense  to  contort  the  data 
to  fit  an  analytical  model  (12).]  This  is  the  reason  descriptive 
statistics  were  used  instead  of  some  of  the  higher-powered  statistical 
models.  However,  cluster  analysis  was  conducted  to  determine  if  there 
existed  a  real  need  to  perform  such  analyses.  Recall  that  in  the 
cluster  analysis,  one  was  looking  for  subsets  of  the  population  that 
scored  very  similar  to  one  another,  but  different  from  the  rest  of  the 
population.  If  such  subgoups  bad  existed,  this  would  have  provided 
good  indication  that  there  were,  indeed,  nonobserved  independent 
variables  that  were  driving  the  scoring  patterns.  The  results  of  the 
cluster  analysis  made  it  quite  clear  that  there  were  no  such  sub- 


groups,  and  therefore,  no  independent  variables  that  were  crucial  to 
the  understanding  of  the  scoring  patterns.  [Such  independent  vari¬ 
ables  could  have  been  age,  income,  educational  background,  military 
service  branch,  sex,  geographical  location  of  domicile  and  birth, 
etc .  ] 

In  summary,  it  was  found  that  descriptive  statistics  provided  suffi¬ 
cient  visibility  into  "who  scored  what"  and  that  there  was  no  statis¬ 
tical  basis  for  determining  "why." 

2.  Long-term  Effects. 

There  exists  a  library-full  of  stochastic  models  that  could  be  used  to 
attempt  to  model  the  long-term  effect  of  the  program  training  on  the 
participants.  In  fact,  in  the  first  year's  report,  the  attempt  was 
made  to  fit  a  stationary  (and  then  a  non-stationary)  Markov  model  to 
the  growth  in  competence.  Data  was  first  manipulated  and  transformed 
from  raw  scores  to  a  simple  set  of  high,  medium,  and  low  scores.  The 
transition  probabilities  that  an  individual  respondent  would  move  from 
one  category  (high,  medium,  or  low)  to  another  over  the  program  and 
then  in  six  month  intervals  were  computed.  While  it  was  possible  to 
model  the  first  three  sample  points  (pre-program,  post-program,  and 
six  months  post-program),  and  even  the  fourth  (12  month  post-program) 
with  a  sufficiently  complex  model,  the  model  was  of  little,  if  any, 
predictive  value.  From  the  data,  it  was  impossible  to  determine  when 
the  growth  process  would  reach  a  steady  state,  and  even  what  the 
nature  of  that  steady  state  would  be.  In  effect,  the  problem  was  to 
choose  the  "best  fit"  to  three  points;  however,  this  fit  did  not 
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provide  a  good  approximation  of  the  fourth  point.  The  basic  problem 
was  one  of  attempting  to  extrapolate  approximately  100%  beyond  the 
total  time  span  of  the  data  sample.  This  is  a  frequent  downfall  of 
time-series  modelling:  forecasting  beyond  the  sample.  When  there  are 
long  time  span  samples  (long  relative  to  the  required  forecast  period) 
then  time-series  forecasting  using  Markov  (12)  models  (as  well  as 
other  models  such  as  Box-Jenkins ,  ARIMA,  semi-Markov,  Power  Spectrum, 
etc.)  is  reasonable.  The  samples  in  this  case  are  just  too  short-term 
to  provide  reliable  indication  about  the  long-term.  The  analysis  that 
was  performed  is  attached  for  the  reader's  information. 

3.  Applicant  Prediction. 

In  attempting  to  predict  the  future  population  of  applicants  to  the 
program,  it  was  postulated  that  the  application  process  within  each 
organization  that  was  a  potential  supplier  of  candidates  was  the 
result  of  a  Poisson  process  with  an  average  application  rate,  X,  for 
each  organization.  Furthermore,  \  was  assumed  to  be  a  linear  function 
of  the  then  extant  recruiting  policies  (number  of  brochures  distribu¬ 
ted,  number  of  visits  by  the  program's  executive  director,  Service’s 
budget  for  such  training,  etc.)  Finally,  it  was  hypothesized  that  the 
final  pool  of  applicants  resulted  from  a  Gamma  distribution.  (Over¬ 
all,  this  meant  that  the  application  process  was  modeled  as  a  random 
negative  binomial  process.)  The  approach  proved  quite  realistic  with 
the  first  year's  data  fitting  the  model  rather  well  (1,4). 
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Formally,  the  model  was  stated  as  follows: 

A'  Ai  =  Po  +  PlXli  +  P2X2i  +  P3X3i  +  P4X4i 
where  A^  =  #  of  applicants  from  agency  i 
=  training  budget  for  agency  i 
X2i  =  ^  e^^8ible  candidates  for  agency  i 

X3i  =  ^  brochures  mailed  to  agency  i 

x^  =  #  of  visits  to  agency  i 
0  =  regression  coefficients 


B.  Probability  that  (#  attend  from  agency  i  =  k)  =  -1  , — 

C.  A.  is  distributed  as  a  Gamma: 

1 


if  A  >  0 

if  A  =  0 


D.  Resulting  in  the  Negative  Binomial: 


Prob  (K=n)  =  qVpn(-l)n(~V) 
where  p  =  and  q  =  1-p 

2 

and  mean  p  =  vp/q  and  variance  o  =  p(l+p/v) 

While  the  first  year's  data  when  used  with  this  model,  as  well  as  a 
simple  Poisson  model,  fit  the  sample  distribution  rather  well,  the 
second  year's  data  did  not  fit  the  established  model  (A  above)  at  all. 
Thus,  the  changes  in  recruiting  approach  from  one  year  to  the  next 
preempted  the  model's  usefulness  as  a  predictor  of  future  applicants. 
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SUMMARY 


Summarizing  the  results  of  the  analysis  of  the  1980  data,  we  conclude 
as  we  did  in  1979  that,  foremost,  the  executive  program  clearly  raises  the 
perceived  level  of  competence  among  all  who  participate,  independent  of  the 
participant's  background  (in  terms  of  military  vs.  civilian  and  War  College 
vs.  non-War  College).  The  trend  over  the  first  three  years  toward  reduced 
pre-program  levels  of  competence  has  been  largely  matched  by  an  offsetting 
trend  toward  increased  program  gains,  indicating  a  certain  degree  of  pro¬ 
gram  effectiveness  in  raising  all  groups  to  a  similar  level  of  competence. 
It  has  been  further  demonstrated  that  the  separate  elements  of  the  program 
tend  to  complement  one  another.  While  it  has  also  been  shown  that  the 
program  curriculum  is  not  directed  toward  any  specific  group  of  partici¬ 
pants,  the  military  and  war  college  attendees  have  characteristically 
responded  in  a  more  consistent  manner  to  the  full  spectrum  of  program 
training. 
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Attachment  A 


NEW  CASE  DEVELOPMENT  FOR  EXECUTIVE  PROGRAM 


Title 

Faculty 

1. 

Dienbienphu 

May 

2. 

Bay  of  Pigs 

May 

3. 

Panama  Canal  Treaty  Negotiations  (A)  and  (B) 

Raif fa/ Johnston 

4. 

United  States  -  Phillipines  Military  Base 
Negotiations 

Raif fa/ Johnston 

5. 

Eurocommunism  (A)  and  (B) 

Yergin/ Johnston 

6. 

Manpower  Management  in  the  Navy  (A)  and  (B) 

Johnston 

7. 

Middle  East  Negotiations:  The  Camp  David  Summit 

Raif fa 

8. 

Congress  and  the  F16 

Moore/Zimmerman 

9. 

Energy  Exercise 

Yergin 

10. 

The  Defense  Budget 

Lynn 

11. 

East  Asian  Security 

Nacht 

12. 

European  Security 

Nacht 

13. 

AWACS  (A)  and  (B) 

Johnston 

14. 

Defense  Management 

Lynn 

15. 

The  Intelligence  Process:  Operation  Barbarossa 

May 

16. 

Political  Instability 

Nacht 

17. 

The  Cruise  Missile  and  NATO  Standardization 

Johnston 

18.  The  President,  the  Congress,  and  the  Nuclear 
Carrier  (A)  and  (B) 


Johnston 


Attachment  B 


DAY 

(Aug  11) 

Aug  12 

Aug  13 

Aug  14 

Aug  15 

Aug  16 

(Aug  18) 

Aug  19 

Aug  20 

Aug  21 

Aug  22 


SCHEDULE  OF  CLASSES 


PERIOD 


SECT  A  (Room  150) 

Bay  of  Pigs/Cuban  Crisis  (May) 
Decision  Analysis  (Raiffa) 

C5A  (Lynn) 


Dienbienphu  &  Pleiku  (May! 
Bargaining  Theory  (Raiffa) 

Defense  Budget  (Lynn) 


SECT  B  (Room  130) 

Combined  Session 
!  C5A  (Lynn) 

I  Decision  Analysis 
(Raiffa) 

Defense  Budget  (Lynn) 
!  Dienbienphu  &  Pleiku 
(May) 

I  Bargaining  Theory 
(Raiffa) 


International  Economics  I  (Uyterhoeven)  -  Combined  Session 
Middle  East  (May/Nacht)  -  Combined  Session 
International  Economics  II  (Uyterhoeven)  -  Combined  Session 

6:30  p.m.  Panel  Discussion:  The  Policymaker  and  Intelligence  (Rm  140) 


Congress  and  the  FI 6  (Moore) 
Media  1  (Miller) 

SALT  I/I I  (Carnesale) 


Organization  Analysis  (Porter) 

Media  II  (Miller) 

International  Negotiations  (Raiffa) 


Food  (Me Craw) 
Grain  (Porter) 


1  SALT  I/II  (Carnesale) 
Combined  Session(Rm  140) 
3  Congress  and  the  FI 6 
(Moore) 

1  International  Negotia¬ 
tions  (Raiffa) 

Combined  Session 
3  Organization  Analysis 
(Porter) 

Combined  Session 
Combined  Session 


SECT  A  (Room  130) 


SECT  B ( Room  150) 


1  Marxism/ Communism  (May)  -  Combined  Session 

2  Strategic  Forces  &  Doctrine  (Carnesale)  2  E.  Asian  Security 

(Nacht) 

3  E.  Asian  Security  (Nacht)  3  Strategic  Forces  & 

Doctrine  (Carnesale) 

1  Proliferation  (Carnesale)  -  Combined  Session 

2  Nuclear  Exports  (Nye)  2  Energy  I  (Uyterhoeven) 

3  Energy  I  (Uyterhoeven)  3  Nuclear  Exports  (Nye) 

5:00  -  6:30  p.»\  Panel  Discussion:  Executive/Legislative  Relations  (Rm  150) 


Proliferation  (Carnesale) 
Nuclear  Exports  (Nye) 
Energy  I  (Uyterhoeven) 


Energy  II  (Uyterhoeven) 

Energy  Security  (Nye) 

Force  Planning  (Johnston) 

International  Economics  II I (Uyterhoeven) 
Dealing  with  Conflict  (Fisher) 

European  Security  (Nacht) 


Energy  Exercise  (Nye) 
Urapup  (Johnston) 


1  Force  Planning  (Johnston) 

2  Energy  II  (Uyterhoeven) 

3  Energy  Security  (Nye) 

-  Combined  Session 

2  European  Security 

(Nacht) 

3  Dealing  with  Conflict 

(Fisher) 

-  Combined  Session 

-  Combined  Session 
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FACULTY 


Graham  T.  Allison  (KSG) 

(617)  495-1122 

Ernest  R.  May  (KSG) 

(617)  495-1144 

James  E.  Austin  (HBS) 

(617)  495-6597 

Thomas  K.  McCraw  (HBS) 

(617)  495-6513 

Jonathan  Brock 
(206)  543-4920 

Arthur  R.  Miller  (HLS) 

(617)  4954111 

Albert  Carnesale  (KSG) 

(617)  495-1189 

Jonathan  Moore  (KSG) 

(617)  495-1360 

Antonia  H.  Chayes  (KSG) 

(617)  495-1149 

Michael  L.  Nacht  (KSG) 

(617)  495-1405 

John  T.  Dunlop  (HBS) 

(617)  495-1431 

Joseph  S.  Nye,  Jr.  (KSG) 
(617)  495-1148 

Roger  Fisher  (HLS) 

(617)  495-4615 

Roger  B.  Porter 
(202)  456-6722 

Stephen  B.  Hitchner 
(202)  633-2661 

Don  K.  Price  (KSG) 

(617)  495-1315 

Douglas  M.  Johnston,  Jr.  (KSG) 

(617)  495-1331 

Howard  Raiffa  (KSG) 

(617)  495-6289 

William  W.  Kaufmann  (MIT) 

(617)  253-3376 

Hugo  E.R.  Uyterhoeven  (HBS) 
(617)  495-6622 

Laurence  E.  Lynn,  Jr.  (KSG) 

(617)  495-1323 

Daniel  H.  Yergin  (KSG) 

(617)  495-1359 
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Attachment  E 


Ratio  scale  ratings  relating  to 
the  perspectives  and  roles  of  other 
participants  in  the  national  security  process 

(PLEASE  READ  CAREFULLY!) 

Please  evaluate  how  competent  you  feel  in  dealing  with/operating 
in  each  of  the  listed  areas.  If  you  feel  completely  comfortable  about 
operating  in  an  area  with  which  you  are  very  familiar,  a  rating  of 
1,000  would  be  appropriate.  If  you  only  feel  "half  comfortable",  then 
the  rating  becomes  500.  If  you  feel  one-tenth  as  comfortable  (as  com¬ 
pared  to  a  very  well-known  area)  ,  then  the  rating  should  be  100.  Ratings 
of  zero  (0)  and  negative  ratings  should  not  be  used.  Please  assign  a 
separate  score  to  each  Roman  numeraled  section  (not  the  individual  ques¬ 
tions)  for  your  pre-program  and  immediate  post-program  evaluations.  This 
questionnaire  will  be  returned  to  you  in  six  months  for  similar  evaluation 
at  that  time. 

The  questions  listed  within  each  section  are  intended  to  be  represen¬ 
tative  of  the  types  of  things  you  should  consider  in  deciding  how  comfor¬ 
table  you  are.  They  should  not  be  interpreted  as  representing  an  exhaustive 
treatment  of  any  given  area. 


Following 


RATIO  SCALE  RATINGS 


B 

u 

01 

4J 

to 

m 

H 

u 

bC 

W 

o 

0 

u 

E 

P-) 

s£> 

1 .  The  Presidency 

-  To  what  extent  dr  Fr«.si dents  embody  rational  purpose  and  policy? 
To  what  extent  are  they  leaders  of  parties  or  factions? 

To  what  extent  do  they  act  just  for  themselves?  How  can  you  tell 
Can  they? 

-  How  do  Presidents  make  decisions?  Do  they  actually  make  them? 
Where  do  they  get  information:  Given  time  constraints,  how  do 
they  choose  among  competing  presentations?  How  do  they  allocate 
scarce  time?  How  do  they  determine  priorities;  or  are  their 
priorities  determined  by  other  people? 

-  How  do  Presidents  differ?  How  important  is  the  President’s  own 
background?  How  important  are  campaigns  —  promises,  etc.? 

How  are  Presidents  different  during  and  after  transitions?  .  .  . 
in  second  terms?  'What  is  the  range  of  potential  differences  in 
operating  style? 

-  Who  else,  if  anyone,  thinks  Presidentiallv?  How  can  cr  do 
Presidents  deal  with  the  fact  chat  Blmost  everyone  else  has  a 
more  compartmented  outlook? 

-  Is  the  President  just  the  big  chief?  Are  his  problems  the 
problems  of  any  executive  or  leader,  or  are  they  different  in 
kind  from  problems  of  a  cabinet  officer,  a  corporation  head,  etc 

-  To  what  extent  does  the  president’s  organization  of  his  White 
House  staff  affect  policy  formulation  and  implementation? 


11 .  The  Congress 

-  What  are  the  xiuances  of  the  legislative  process  that  might  make 
a  difference  in  the  way  one  frames  one's  programs  prior  to 
submission  for  Congressional  approval? 


-  What  are  the  priorities  that  are  likely  to  be  accorded  com¬ 
peting  demands  for  resources  by  the  various  national  security- 
related  Congressional  committees? 

-  On  what  issues  is  Congressional  decision-making  likely  to  be 
driven  by  the  national  interest,  as  opposed  to  local  political 
considerations? 

-  How  should  one  deal  with  the  situation  where  it  looks  likely 
that  the  substance  of  the  issue  may  be  subsumed  in  jurisdic¬ 
tional  infighting,  i.e.,  protection  of  tuTf  vis-a-vis  the 
Executive  Branch,  between  committees,  etc." 

-  To  what  extent  should  one  provide  one's  personal  opinion  vis-V 
vis  total  support  for  the  Administration  position? 
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III.  Political  Appointees 

-  How  should  one  deal  with  the  fact  that  the  terns  and  conditions 
of  employment  of  political  appointees  are  considerably  dif¬ 
ferent  from  those  for  other  categories  of  participants  in  the 
nalonal  security  process? 

-  What  motivates  them? 

-  How  does  their  different  risk  orientation  affect  their 
behavior  and  perception  of  things? 

-  What  is  the  best  way  to  cope  with  the  managerial  shortcomings 
of  the  various  categories  of  political  appointees?  i.e., 
educators  often  are  not  skilled  in  the  management  of  large 
organizations;  lawyers  are  often  not  skilled  In  either 
quantitative  analysis  or  the  administration  of  large  organiza¬ 
tions;  and  businessmen  often  have  trouble  operating  in  a 

"no  bottom  line"  environment. 

-  In  what  sorts  of  circumstances  are  political  considerations 
likely  to  prevail  over  those  of  substance? 


IV.  Careerists 


-  To  what  extent  does  the  career  civilian’s  time  hori.ron 
Influence  his  or  her  approach  to  substantive  decision¬ 
making? 

-  How  should  the  career  military  man  reconcile %the  often 
competing  demands  of  loyalty  to  Service  vis-a-vis  the 
requirement  to  take  direction  from  civilian  officials? 

-  What  are  the  differences  in  planning  priorities  between 
military  and  civilian  planners? 

-  What  are  the  effects  of  frequent  turnover  of  military 
personnel  on  program  decision-making  and  accountability? 

-  To  what  extent  does  a  "civil  service  mentality"  Impede 
creativity  and  initiative  in  program  formulation  and 
implementation? 
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V.  Media 


-  How  does  one  reconcile  the  inherent  conflict 
between  the  media's  interpretation  of  obligations 
and  rights  under  the  First  Amendment  and  the  need 
for  secrecy  that  often  underlies  "national  security 
considerations"? 

-  What  motivates  the  press  in  its  pursuit  of  news? 

-  How  can  one  use  the  media  to  help  achieve  one's 
own  objectives? 

-  How  does  one  avoid  press  chicanery,  such  as  the 
slanting  of  stories  to  achieve  sensationalism  and 
quoting  out  of  context? 

-  What  is  the  language  with  which  one  should  be  familiar 
when  dealing  with  the  media  ("off  the  record",  "deep 
background",  "non-attribution",  etc.)? 

-  Is  citing  a  "personal  opinion"  ever  appropriate  for 
a  top  level  official  when  dealing  with  the  media? 


VI.  Private  Sector 


-  What  are  the  principal  differences  between  public 
and  private  sector  operating  environments? 

-  What  capabilities  does  an  Administration  have  for 
orchestrating  the  disparate  actions  of  multiple  pri¬ 
vate  entities  in  the  national  interest  (such  as  the 
sale  of  wheat  by  U.S.  farmers  to  other  countries)? 

-  To  what  extent  can  the  U.S.  influence  major  inter¬ 
national  concerns  (such  as  the  major  oil  com¬ 
panies? 

-  How  are  trade-offs  calculated  when  different  values 
are  involved  (such  as,  when  the  price  of  more  jobs 
Includes  a  dirtier  environment)? 
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VII.  International 


-  In  negotiating  with  a  foreign  government  or  in  mediating 
between  foreign  governments,  what  approach  can  be  taken 
to  maximize  the  likelihood  of  working  out  terms  that  are 
mutually  advantageous  (as  opposed  to  terms  that  repre¬ 
sent  victory  for  one  party  and  defeat  for  the  other)? 

-  When  negotiating  an  agreement  with  representatives  from 
another  country,  to  what  extent  is  it  necessary/ 
desirable  to  understand  not  only  their  nation's  real 
interests  but  the  historical  perspective  in  which  they 
see  the  issues?  Can  it  be  done?  Axe  there  situations  in 
which  it  would  be  better  not  to  know  at  all  rather  than 
to  know  only  a  little" 

-  What  are  some  of  the  key  determinants  of  differences  in 
national  perspectives? 

-  What  are  some  of  the  principal  differences  between 
American  and  Soviet  approaches  to  strategic  arms  control? 
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Attachment  G 


INTRODUCTION  TO  THE  T-TEST  OF  SIGNIFICANCE 

In  many  investigations  the  researcher  is  primarily  interested  in  discovering  and  evaluating 
differences  between  effects,  rather  than  the  effects  themselves.  For  example,  one  may  be 
interested  in  the  difference  in  “income  for  people  at  various  levels  of  education.  The  most 
common  of  this  type  of  analysis  is  the  comparison  of  two  groups  of  subjects,  with  the  group 
means  as  the  basis  for  comparison.  An  example  of  this  would  be  to  determine  the  difference  in 
income  between  college  graduates  and  nongraduates.  This  example  is  an  instance  where  the  two 
groups  preexist  the  analysis.  In  some  cases,  a  researcher  may  randomly  assign  subjects  to  two 
croups  and  apply  a  treatment  to  one  group.  Treatment  effects  are  measured  by  comparing  the 
two  groups.  For  example,  the  effect  of  a  brand  of  toothpaste  on  the  prevention  of  cavities  might 
be  tested  this  way. 

In  the  comparison  of  group  means,  the  term  treatment  is  used  to  refer  to  the  basis  on 
which  the  two  groups  are  differentiated.  In  the  first  example,  college  education  is  the  treatment; 
in  the  second  example,  it  is  very  natural  to  call  the  toothpaste  the  treatment. 

Statistical  Inference  from  a  Sample  of  Cases 

Since  it  is  most  often  impossible,  or  at  least  impractical,  to  compute  a  group  mean  based 
on  all  members  of  the  group,  the  researcher  must  use  a  sample.  The  true  but  unknown  mean  for 
a  group  is  called  the  population  mean;  it  is  estimated  by  the  sample  mean.  The  comparison  of 
two  group  means  is  thus  a  problem  of  comparison  of  two  sample  means,  and  from  that,  inferring 
the  difference  between  the  means  of  the  parent  populations. 

The  basic  problem  is  to  determine  whether  or  not  a  difference  between  two  samples 
implies  a  true  difference  in  the  parent  populations. 

Since  it  is  highly  probable  that  two  samples  from  the  same  population  would  be  different 
due  to  the  natural  variability  in  the  population,  it  is  clear  that  a  difference  in  sample  means  does 
not  necessarily  imply  that  the  populations  from  which  they  were  drawn  actually  differ  on  the 
characteristic  being  studied. 

The  goal  of  the  statistical  analysis  is  to  establish  whether  or  not  a  difference  between  two 
'amples  is  significant.  “Significant”  here  does  not  mean  “important  or  “consequence  ;  it  is 
used  here  to  mean  “indicative  of  or  '•signifying"  a  true  difference  between  the  two  populations. 
The  systematic  approach  used  to  test  sample  differences  is  as  follows. 

1  A  null  hypothesis  and  a  corresponding  alternative  hypothesis  are  formulated.  The  null 
hypothesis  (Hu)  must  be  a  precise  statement  for  which  the  Student’s  t  statistic  ($nd 
probability)  can  be  computed. 

Typically.  //«  is  what  the  researcher  is  trying  to  disprove  or  reject  so  that  the  alternative 
hypothesis  (//,)  can  be  accepted.  Most  often  H0  states  that  the  population  means  are  the  same 
(H0:  Another  possible  statement  for  H0  is  that  the  population  means  differ  by  » 

specific  amount,  for  example.  H,  is  usually  the  set  of  all  other  possible 

outcomes  (for  example,  //,:/Xi^Mj  or  H 

2  A  “significance  level”  for  testing//,,  is  chosen.  Since  sampling  is  being  used,  a  decision  to 
accept  or  reject  H0  cannot  be  made  with  absolute  certainty;  the  decision  must  be  based  on 
probabilities.  The  significance  level  is  the  smallest  probability  that  will  be  accepted  as 
reasonable,  i.e.,  due  to  chance  or  sample  variability. 

3  The  samples  are  taken  and  the  two  sample  means  and  variances  are  computed. 

4  Assuming  Ha  is  true,  the  t  statistic  (see  below)  is  computed.  From  the  frequency  distribution 
of  the  statistic  is  computed  the  probability  of  getting  a  more  exrreme  value  of  the  statistic. 
Intuitively,  this  is  the  probability  of  drawing  two  samples  that  differ  more  than  the  pair 
actually  drawn. 

5  If  the  probability  computed  in  step  4  is  smaller  than  the  significance  level  chosen  in  step  2,  Hn 
is  rejected.  If  the  probability  is  larger,  W0  is  not  rejected.  However,  this  does  not  necessarily 
imply  that  //„  is  true,  only  that  the  true  situation  is  not  significantly  different  from  that 
assumed  by  the  null  hypothesis. 


1  lik’  T-TEST  procedure  *a>  developed,  programmed,  and  documented  for  the  CDC  version  of  SPSS  by  James  Tuccv 
"l  'v'fihweMcrn  I'nivcrvrlv 


Typical  values  for  the  significance  level  chosen  in  step  2  are  .05  or  .01 .  The  specific  value 
of  the  significance  level  chosen  is  based  on  the  seriousness  of  the  type  I  error  (rejecting  H0  when 
it  is  true)  as  opposed  to  type  II  error  (accepting  //0  when  it  is  false).  The  significance  level  is 
exactly  the  probability  of  rejecting  H0  when  it  is  true.  Thus,  if  type  I  error  is  very  serious,  the 
significance  level  would  be  set  correspondingly  low  (.001  is  sometimes  used).  On  the  other 
hand,  if  type  II  error  has  the  worse  consequence,  the  significance  level  could  be  raised,  e.g., 
.10. 


The  t  Distribution 

Student's  t  is  the  statistic  used  in  calculating  the  probability  associated  with  Hu  The  t  is  a 
statistic  generally  applicable  to  a  normally  distributed  random  variable  where  the  mean  is  known 
(or  as  we  shall  see,  assumed  to  be  known)  and  the  population  variance  is  estimated  from  a 
sample.  Assume  that  the  normally  distributed  Tandom  variable  X  has  mean  /x  and  unknown 
population  variance  cr which  is  estimated  by  a  sample  variances2.  Then,  t  =  ( X-fi)s .  Note 
that  this  formula  is  almost  identical  to  that  for  the  standard  normal  deviate  z  —  (X— fx)/<r. 
Historically,  the  statistic;  was  always  used  for  computing  the  probability  of  occurrence  for  normal 
variables,  and  tables  of  such  probabilities  were  constructed  and  used  for  that  purpose.  The 
statistician  William  Seeley  Gosset  (who  used  the  pseudonym  “Student")  realized  that  if ;  were 
computed  using  s  based  on  a  small  sample,  the  use  of  normal  tables 'wai  not  trustworthy  and  an 
alternative  table  was  needed.  The  t  distribution  is  a  direct  result  of  his  work,  thus  the  name 
Student's  t.  The  t  distribution  depends  on  the  degrees  of  freedom  used  in  computings.  This  is 
the  denominator  in  the  s2  calculation  below  and,  in  the  case  of  one  sample  of  size  n.  is  n  -  1 
where 


» 


t  is  tabulated  for  various  degrees  of  freedom,  usually  from  1  to  30.  For  degrees  of  freedom 
larger  than  30.  s2  (the  sample  variance)  is  a  sufficiently  reliable  estimate  of  a2  (the  population 
variance)  so  that  the  distribution  of  t  is  almost  identical  to  that  of This  is  indicated  ip  t  tables 
by  the  entry  *■  (infinity)  for  degrees  of  freedom.  The  probability  given  for  /  is  usually  two-tailed, 
that  is,  the  probability  for  a  value  of  |r  j  (the  absolute  value  of  /)  or  larger. 

The  t  is  a  statistic  which  may  be  computed  for  a  normally  distributed  variable;  to  compute 
a  r  value  for  a  pair  of  sample  means,  the  following  points  must  be  considered: 


1  The  sample  mean  is  a  normally  distributed  variable.  That  is,  given  a  normal  population  with 
mean  /x  and  variance  of  <72,  if  samples  of  size  n  are  drawn,  the  sample  means  are  normally 
distributed  with  mean  /x  and  variance  (<rs/n ). 

2  The  difference  of  two  normally  distributed  random  variables  is  a  normal  random  variable. 
That  is,  given  two  random  variables  X,  and  Xs  with  means  jx,  and  and  variances  a,2  and 
cr?,  respectively,  the  random  variable  D  =  X,  -  X,  generated  by  pairs  independently 
selected  from  the  two  populations  is  normally  distributed  with  mean  nt  -  fi.:  and  variance 
<r,*  +  <r3 2. 

From  the  above,  the  difference  of  sample  means  is  a  normally  distributed  variable  with 
mean  (m  ,  -  /us)  and  variance  (cr  ,2/« ,  +  cr^hu). 


Comparison  of  Means — Independent  Samples,  Populations  with  Com¬ 
mon  Variance 

Given  two  populations  with  means  fi ,  and  /x2,  respectively,  and  common  variance  <rs,all 
unknown,  the  problem  is  to  use  the  /-test  of  significance  to  determine  if  /a,  =  n  ,.  Following  the 
steps  outlined  above: 

1  The  null  hypothesis  ,  =  n>  and  alternative  hypothesis  ^  fi.,  are  formulated. 

2  .The  significance  level  a  is  chosen. 

3  The  two  populations  are  sampled;  means*,  and.Tit  variances-!,2  and s2  are  computed,  based 
on  samples  of  sizes?/ ,  and  n.,,  respectively.  From  the  pair  ot'sample  -variances,  which  are  both 
estimates  of  or2,  the  "pooled  variance"  is  computed. 

2  _  0*1  ~  0*i2  +(»2  ~  ])522 

*  («l  -  l)  +  (/t2  -  0 

[Note  that  there  are  (n,  +  ns  -  2)  degrees  of  freedom  in  the  computation.]  s 2  is  the 
weighted  average  of  the  sample  variances  and  is  the  best  estimator  for  cr2.  Writing  J ~  7,  - 
.7j,  since  the  population  variance  for  the  difference  in  sample  means  is  ad2  =  cr ,2//r ,  + 
cr.:2/n...  the  sample  variance  for  the  difference  of  sample  means  is 

=  (!>!  +S>2) 

/  corresponding  to  the  difference  in  sample  means  is  computed: 


(7,  -*2)-Fi  -A<2 


which,  under  the  null  hypothesis  //„:  (>u ,  =  n>),  reduces  to 

t  -  = -  with  (/tj  +  n-)  -  2)  degrees  of  freedom. 

sj 

4  The  probability  associated  with  /  is  computed.  The  probability  is  for  the  occurrence  of  a  value 

equal  to  or  larger  than  /,  sign  ignored.  This  is  the  twi  tailed  probability  and  it  is  appropriate  to 
the  set  of  hypotheses  chosen  since  they  do  not  assume  that  /  will  be  either  positive  or 
negative.  Tests  of  this- type.  i. e— .where  specifies  inequality,  are  called  two-iailed  Tcmz 

One-tailed  tests  may-also  be  used. -as  described  in  SecT-47.2.^-6r* 

5  H„  is  rejected  if  the  two-tailed  probability  for  H„  is  less  than  a  chosen  in  step  2. 


Comparison  of  Means-fndependent  Samples,  Populations  with  Unequal 
Variances 

Given  populations  with  unequal  variances.  /  cannot  be  computed  for  the  difference  in 
sample  means.  Instead,  an  approximation  to  /  may  be  computed. 


(*]  --*2)  -  (Mi  -  M2) 

t  - - ; - ; - 

S\/n\+s2/nZ 

This  statistic  is  not  distributed  as  Student’s  t.  However,  the  probability  for  t  can  be  approxi¬ 
mated  by  treating  it  as  /,  but  with  degrees  of  freedom 

(Ol2/"l)  +  (*i 2/«2>l 2 

df  =  "I - 7 - 5 - 7 - 7 - 

[(s,2/«l)-/(Wl  -  1)]  +  [(r22/n2)'/(«2  -  1)] 

This  number  is  usually  not  an  integer,  but  a  reasonably  accurate  probability  is  obtained  by 
rounding  to  the  nearest  integer. 

If  it  is  not  known  whether  the  two  populations  have  the  same  variance,  an  F  test  of  sample 
variances  may  be  performed:  The  null  hypothesis  Hn:<T,2  =  err  with  alternative  ^  cr2 

and  a  significance  level  a'  is  chosen  (a'  does  not  necessarily  have  the  same  value  as  a  used  for 
the  /-test).  From  the  sample  variances,  F  is  computed. 

p  _  larger  s 2  ’  “• 

smaller  s2  1  ‘ ' 

If  the  probability  for  F  is  greater  than  a',  H0  is  accepted;  t  based  on  the  pooled-variance 
estimate  for  0752  should  be  issued. 

If  the  probability  forF  is  less  than  or  equal  to  a',  H0  is  rejected;  t  based  on  the  separate 
variance  estimate  for  crn2  should  be  used. 


Attachment  H 


CLUSTER  ANALYSIS 


The  specific  algorithm  used  for  the  cluster  analysis  opera tLS  as 
follows: 

1)  Data  Set:  the  data  used  consists  of  the  7  "during -program" 
gains  for  each  of  30  participants. 

2)  First  compute  the  means,  x^,  and  standard  deviations,  sit 
for  each  area,  where  i  indexes  the  area. 


3)  Standardize  each  participant's  score.  That  is.  If  xi  equals 

a  given  participant's  score  for  area  1,  then  his  standardized 

score  (sometimes  referred  to  as  a  z-score)  for  that  area 
x1“*1  * 

equals  . .  .  This  Is  done  for  each  of  the  seven  areas  for 

51 

all  of  the  cases. 


4)  Compute  the  distance  between  each  case.  The  distance  is  de¬ 
fined  as  follows:  Let  y^,  1*1,  7  be  the  standardized  scores 
for  case  y  and  Zj,  1*1,  7  for  case  z.  Then  the  distance  be¬ 
tween  y  and  z  is  equal  to 


s(yrZir 
1*1  1  1 


This  distance,  d.  Is  computed  for  all  pairwise  combinations 
of  cases. 


5)  Begin  forming  clusters  by  joining  the  two  cases  with  the 

smallest  distance  between  them.  Note  that  this  distance  is  a 


measure  of  how  similar  cases  are;  the  more  alike 
the  cases  (yi -z^  will  be  closer  to  0),  the  smaller  is 
the  distance  between  them.  Where  two  cases  are  joined, 
the  average  of  each  yi  and  zi  [*l/2(yi+z1)]  becomes  the 
standardized  scores  for  the  cluster. 

6)  Update  the  distance  lists  by  replacing  the  distances  from 

each  case  to  cases  y  and  z  with  the  distance  to  the  yz 

cluster. 

% 

7)  Continue  joining  the  cases  into  clusters;  at  each -step 
join  either  the  two  closest  cases,  the  two  closest  clusters 
or  the  closest  case- to -cl us ter  pair,  whichever  is  the 
smallest  distance  overall.  After  -each  joining,  update  the 
distance  lists  so  that  the  new  distances  reflect  the  weighted 
average  of  the  elements  combined  in  the  new  cluster.  (Note: 
keep  track  of  the  distance  between  the  cases  or  clusters 
that  are  joined.) 

8)  Stop  clustering  when  all  cases  are  joined  into  a  single  large 
cluster.  (This  is  a  simple,  straightforward  cluster  tech¬ 
nique  used  in  the  BMDP2M  program  of  the  Biomedical  Computer 
Program  P-Series  (1977)  developed  at  the  Health  Sciences  Com¬ 
puter  Facility,  UCLA.  For  more  complex,  specialized,  albeit 
sophisticated  and  powerful  techniques,  see  Clustering  Algo¬ 
rithms.  Hartigan,  J.A.,  John  Wiley  &  Sons,  New  York,  1975). 


Attachment  H 
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Attachment  I 


CORRELATION  ANALYSIS 


1)  x  and  y  are  the  average  "during-program"  gains  i  areas 
x  and  y. 

2)  x^  and  y^  are  the  gains  of  case  i  in  areas  x  and  y. 

3)  Var  x  and  Var  y  are  the  variances  of  "during-program" 

gains  in  areas  x  and  y.  N 

4)  Using  the  above  definition,  P«  ■ —  - . —  ■  — 

/Var(x)Var(y) 

where  N  is  the  number  of  participants  and  p,the  "cor¬ 
relation  coefficient",  varies  between  -1  and  1.  When  x 
and  y  are  perfectly  correlated,  that  is,  when  x  is  a 
positive  linear  function  of  y,  or  they  both  are  linear 

functions  (with  the  same  sign)  of  some  other  variable  z, 

*  * 

then  P»l.  If  x  is  a  negative  linear  function  of  y,  then 
o«-l.  If  x  and  y  are  uncorrelated,  p«0.  In  general  terms, 
if  p>0,  x  increases  as  y  increases,  and  If  P<0,  x  decreases 
as  y  Increases.  The  closer  to  l.(or  -1)  p  is,  the  more 
closely  x  tracks  changes  in  y. 

The  application  to  the  problem  at  hand  Is  as  folio*?:  If  an  attendee 
gains  a  great  deal  during  the  program  In  area  A,  can  he  or  she  be  expected 
to  gain  significantly  in  area  B? 


